Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve performance quantized networks. In this study, we propose stochastic precision ensemble training QDNNs (SPEQ). SPEQ is a scheme; however, teacher formed by sharing model parameters student network. We obtain soft labels randomly changing bit activation stochastically at each layer forward-pass computation. trained with these reduce noise. cosine similarity loss employed, instead KL-divergence, KD training. As changes continuously random bit-precision assignment, it exploits effect KD. outperforms existing methods various tasks, such as image classification, question-answering, and transfer learning without need cumbersome